Repository navigation
aitools: bound best-effort telemetry auth - #7468
Conversation
0d29efd to
a15150a
Compare
|
Commit: Regression test reportSelected 2 changed tests (limit=20, 0 not selected) 2 tests: regression=2 unreleased=0 coverage=0 failing=0
TestAccept/cmd/aitools/install-host-metadata: branch ✅ | main (0f407e7) ❌ | latest (v1.26.0) ❌main (0f407e7): latest (v1.26.0): TestAccept/cmd/aitools/telemetry/install: branch ✅ | main (0f407e7) ❌ | latest (v1.26.0) ❌main (0f407e7): latest (v1.26.0): |
✅ Integration test reportCommit: d13de92
Top 11 slowest tests (at least 2 minutes):
|
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at a15150a. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 blocking issue(s), tagged [blocking]; 1 open design question.
This PR stops aitools install from blocking on an unreachable workspace. Telemetry auth now resolves in the background on a shadow command, the wait is capped at 3 s from start, and its logs show only with --debug. The approach works as described and CI is green on all platforms. One blocking concern remains: when the wait ends while an OAuth refresh is still running, the CLI exits before saving the new single-use refresh token, so the user can be logged out. Two smaller suggestions are to skip the wait when no event was logged and to add an end-to-end upload test, plus a design question about reusing Upload's existing timeout instead of adding a second one.
Questions
- libs/telemetry.Upload already bounds config resolution and authentication within its own timeout (newAPIClientWithin and authenticate). Did you consider putting an unresolved config (with --profile and env applied) on the context and letting Upload resolve it? Or moving this helper into root or libs/telemetry so other commands can reuse it? Either would remove the shadow command, goroutine handoff, and log handler from cmd/aitools. The trade-offs are less overlap with the install, MustAnyClient's account-client fallback, and where hostmetadata warnings surface. Which approach do you prefer, and why?
Comments outside the changed lines
- [nit]
.github/PULL_REQUEST_TEMPLATE.md: The description uses## Summary,## Evidence, and## Merge Danger..agents/skills/pr-checklist/SKILL.mdasks agent-written PRs to follow.github/PULL_REQUEST_TEMPLATE.mdexactly (## Changes,## Why,## Tests, with no invented sections) and to keep the summary concise. Please move the content under those headings and trim it.
| log.Debugf(sctx, "auth for telemetry panicked: %v", r) | ||
| } | ||
| }() | ||
| resolve(shadow, args) |
There was a problem hiding this comment.
[blocking] The background resolution can refresh a U2M token: PersistentAuth.refresh posts to the token endpoint and then stores the result. The process exits without waiting for it once apply gives up. Example: databricks-cli auth, an expired access token, a slow but reachable workspace, and a short --skills-only install where the host-metadata fetch plus the refresh straddle the 3 s deadline. The server rotates the single-use refresh token (cmd/auth/README.md), the new one is never stored, and the next command fails until databricks auth login. Before this PR the refresh always finished in PreRunE. A best-effort telemetry path should not be able to log the user out. Please keep the refresh out of the goroutine that gets abandoned. For example, resolve only the config in the background and authenticate on the main goroutine in apply once it has resolved, so a reachable host makes this one short request. Otherwise, get an explicit sign-off from the auth owners that this risk is acceptable.
There was a problem hiding this comment.
Fixed in e503a62. The background resolution never refreshes a token now. A new root.SkipTokenRefresh context option makes MustWorkspaceClient/MustAccountClient use auth.CredentialsWithoutRefresh, whose databricks-cli strategy passes the new u2m.WithoutRefresh: PersistentAuth.Token returns the cached access token only while it is valid (it skips the proactive refresh too) and returns ErrRefreshDisabled for an expired one. A failed resolution now leaves no config on the command, so Upload does not retry auth (or refresh) at exit either. An expired token therefore just drops the event, and the next command that authenticates refreshes it as usual.
Tests: TestToken_WithoutRefresh (no token request, nothing stored), TestCLICredentialsConfigure_NoRefresh, TestCredentialChainWithoutRefresh, TestTelemetryAuthFailedResolutionUploadsNothing, and acceptance/cmd/aitools/telemetry/install, which installs with an expired cached token and checks that there is no /oidc/v1/token request and no upload, and that the cache is unchanged. Without SkipTokenRefresh that case shows the refresh POST.
| if p == nil { | ||
| return | ||
| } | ||
| if !p.wait() { |
There was a problem hiding this comment.
[nice to have] apply waits up to the budget even when the run logged no telemetry event, and Upload then returns at len(l.logs) == 0. On an unreachable workspace, install --path ./dir, the flag-validation errors, "no agents found", and "Cancelled." all return before logInstallEvent's defer, yet still sit until 3 s after start for nothing. Return early when len(telemetry.BufferedEvents(cmd.Context())) == 0. That check is correct here because the install event is logged in a defer inside runE. Also update BufferedEvents' doc, which says only tests use it. Returning early also exits sooner while a refresh may be in flight, so this is safest once the refresh no longer runs in the background goroutine.
There was a problem hiding this comment.
Done in e503a62. apply returns before waiting when telemetry.BufferedEvents is empty (covered by TestTelemetryAuthNoEventDoesNotWait), and BufferedEvents' doc now mentions this use. Exiting early can no longer cut a refresh short, since the background resolution never refreshes.
| export DATABRICKS_HOST=https://unreachable.test | ||
|
|
||
| title "workspace unreachable: the background auth resolution prints nothing\n" | ||
| trace $CLI aitools install --skills-only --scope=global --agents=claude-code |
There was a problem hiding this comment.
[nice to have] The success path, where the real MustAnyClient runs on the shadow command and the copied config reaches root's Upload, is covered only by a unit test with a fake resolver. If the shadow stopped seeing --profile, or root uploaded from a different context, aitools install would silently stop uploading its event and no test would fail. Please add a case under acceptance/cmd/aitools/telemetry/, which already records /telemetry-ext and has print_telemetry. It would run aitools install --skills-only --scope=global --agents=claude-code against the mock server, with this test's manifest stubs, and print the uploaded aitools_install_event.
There was a problem hiding this comment.
Added acceptance/cmd/aitools/telemetry/install in e503a62. It runs aitools install --skills-only --scope=global --agents=claude-code against the mock server and prints the uploaded aitools_install_event for PAT auth and for a databricks-cli profile passed with --profile and a valid cached token. A third case uses an expired cached token and checks that nothing is refreshed or uploaded.
While there: the new base includes #7003 (title adds its own trailing newline), so I dropped the \n from this script's titles, which had made the test fail on the restacked head.
| // telemetryAuthBudget bounds how long a command waits, at the end of its run, | ||
| // for the best-effort auth resolution that lets its telemetry upload. The | ||
| // resolution runs in the background while the command works, so the command | ||
| // waits only for the part of it that outlasts its run, and only until this | ||
| // budget after its start; a resolution still unfinished at the later of the | ||
| // end of the run and that deadline drops the events. Matches the upload's own | ||
| // default timeout. A resolution that does finish is followed by the upload, | ||
| // which has its own timeout, so after a short run a slow but reachable | ||
| // workspace can add up to about twice this. | ||
| // | ||
| // While the workspace stays unreachable, every run takes at least this long | ||
| // (the budget starts when the command does, so a run that takes longer waits | ||
| // nothing at the end), including runs that buffered no event and upload | ||
| // nothing, such as a flag-validation error: the SDK retries the host-metadata | ||
| // request on connection errors, so the fetch outlasts the budget and the | ||
| // process exits before libs/hostmetadata writes its negative-cache entry, and | ||
| // the next run probes the host again. | ||
| var telemetryAuthBudget = 3 * time.Second |
There was a problem hiding this comment.
[nit] The doc comments on telemetryAuthBudget (lines 21-37) and startTelemetryAuth (lines 70-87) are each about 17 lines and re-derive the timing and risk analysis from the PR description. .agents/rules/comments.md asks for concise doc comments that give the contract and any surprising behavior, not a re-spec. Please trim each to a few lines: what the budget bounds and that an unfinished resolution drops the event, and why the shadow command and the debug-only logger exist. The risk matrix can stay in the PR description.
There was a problem hiding this comment.
Trimmed both in e503a62. telemetryAuthBudget now states what it bounds and that an unfinished resolution drops the events. startTelemetryAuth covers why the shadow command and the debug-only logger exist and that it never refreshes a token. The timing and risk analysis stays in the PR description.
| } | ||
| }() | ||
| resolve(shadow, args) | ||
| if sctx := shadow.Context(); cmdctx.HasConfigUsed(sctx) { |
There was a problem hiding this comment.
[nit] if sctx := shadow.Context(); ... declares a second sctx inside the goroutine, while the deferred recover above logs on the outer one. Two contexts under one name invite a later edit to use the wrong one. Please rename it (e.g. resolved) or call shadow.Context() directly.
2ca5f55 to
0326a2e
Compare
a15150a to
86362f1
Compare
0326a2e to
673bfd5
Compare
86362f1 to
3147276
Compare
673bfd5 to
e1c80b3
Compare
3147276 to
920dbc0
Compare
e1c80b3 to
71b4c1a
Compare
95aeec8 to
e503a62
Compare
|
Thanks for the review. The inline threads have replies; here are the two review-level points. Question 1 (resolving in
Generalizing it into root or PR template nit: this PR is part of a stack whose bodies all use one format (Summary / Evidence / Merge Danger). I'd rather not rewrite all of them, unless you feel strongly about it. |
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at e503a62. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 2 open design questions.
This PR stops aitools install from blocking on an unreachable workspace. Telemetry auth now runs in the background and is waited for at most 3 s, its logs show only with --debug, and it never refreshes an OAuth token. All earlier review comments are addressed, including the abandoned-refresh risk, and CI is green on Linux, macOS and Windows with the new tests running on all three. Two design questions remain: dropping install events for users whose cached U2M token has expired, and whether a bounded synchronous resolution would be simpler. The rest are small doc and test nits.
Questions
- With SkipTokenRefresh, a databricks-cli user whose cached access token has expired (these last about an hour) uploads no aitools_install_event. The base branch refreshed in PreRunE and uploaded it. Since aitools install often runs on its own, how large and how skewed do you expect that share to be? Would you accept it and state the expected share under Merge Danger, or authenticate with refresh on the main goroutine in apply once the background resolution has resolved the host, so a reachable host costs one token request at exit?
- Your earlier reply explains why you prefer the shadow command over resolving in Upload. Did you also consider keeping the synchronous PreRunE resolution but bounding it with the SDK's HTTPTimeoutSeconds and RetryTimeoutSeconds, as cmd/auth/profiles.go already does for unreachable hosts? The host-metadata fetch goes through the SDK client that honors both. That would keep normal token refresh, and so the expired-token events, and remove the goroutine, shadow command, debug-only handler and the new no-refresh mode. The costs are that auth no longer overlaps with the install and the hostmetadata warning stays visible. Which trade-off do you prefer?
Comments outside the changed lines
- [nit]
cmd/aitools/telemetry.go: PR description: Evidence says "Tests at head, all pass (Linux only)", and Merge Danger lists "Untested: Windows and macOS (the 3 s timing, goroutines at process exit)". CI'stask test (macos, direct)andtask test (windows, direct)jobs ran and passedTestTelemetryAuth*,TestToken_WithoutRefresh,TestAccept/cmd/aitools/install-host-metadataandTestAccept/cmd/aitools/telemetry/install. Please update both sections so reviewers don't count a risk that CI already partly covers.
Agreeing with existing threads
- [nit] PR body does not follow .github/PULL_REQUEST_TEMPLATE.md: #7468 (review)
| return ok && skipLoadBundle | ||
| } | ||
|
|
||
| type skipTokenRefresh int |
There was a problem hiding this comment.
[nit] type skipTokenRefresh int goes against .agents/rules/style-guide-go.md, which says to use empty struct types for context keys and lists type myKeyType int as BAD. The neighbouring skipPrompt and skipLoadBundle keys use the old pattern, so this only affects the new key. Suggest type skipTokenRefreshKey struct{} with context.WithValue(ctx, skipTokenRefreshKey{}, true).
There was a problem hiding this comment.
Changed to type skipTokenRefreshKey struct{} in 76d9ee3. The neighbouring keys are left as they are.
| // SkipTokenRefresh makes [MustWorkspaceClient] and [MustAccountClient] use a | ||
| // cached OAuth token only while it is valid and fail instead of refreshing it. | ||
| // It is for best-effort auth that the command may abandon at exit, where an | ||
| // unfinished refresh could lose the server's rotated single-use refresh token. | ||
| // It does not reach a bundle's workspace client; combine it with | ||
| // [SkipLoadBundle]. |
There was a problem hiding this comment.
[nit] The doc says SkipTokenRefresh makes MustWorkspaceClient and MustAccountClient fail instead of refreshing, and names only the bundle exception. The interactive profile pickers build a fresh databricks.Config{Profile: profile} without CredentialsWithoutRefresh: workspaceClientOrPrompt at cmd/root/auth.go:352 and accountClientOrPrompt at :170. A caller that sets SkipTokenRefresh without SkipPrompt can therefore still refresh, and lose the rotated refresh token, on an interactive terminal. aitools sets SkipPrompt, so nothing is wrong today. Please say in the doc that it must also be combined with SkipPrompt, or apply the no-refresh credentials to the picked-profile config.
There was a problem hiding this comment.
Documented in 76d9ee3. The doc now says SkipTokenRefresh applies only to the config that MustWorkspaceClient/MustAccountClient build, that a bundle's workspace client and an interactively picked profile still refresh, and that callers must combine it with SkipLoadBundle and SkipPrompt. aitools sets both.
| // CredentialsWithoutRefresh returns the CLI credentials chain with a | ||
| // databricks-cli strategy that uses the cached OAuth token only while it is | ||
| // valid and never refreshes it (see [u2m.WithoutRefresh]). For best-effort | ||
| // auth that the command may abandon at exit, such as telemetry's. |
There was a problem hiding this comment.
[nit] With no auth_type (for example DATABRICKS_HOST from the environment, or a hand-written profile), the SDK chain ignores each strategy's dry-run error and tries the next one (credentialsChain.Configure). An expired cached U2M token now fails the databricks-cli dry run with ErrRefreshDisabled, so the chain moves on to the metadata-service, OIDC, Azure and Google strategies. For example, on an Azure host with az logged in, the background goroutine runs az account get-access-token and uploads telemetry as the az identity. The default chain would have refreshed and stopped at databricks-cli. That is acceptable for best-effort telemetry, but please mention it in this doc comment and in Merge Danger so later callers of CredentialsWithoutRefresh know.
There was a problem hiding this comment.
Documented in 76d9ee3 on CredentialsWithoutRefresh. With auth_type = databricks-cli an expired token fails auth. Without an auth_type, the chain skips the failed dry run and moves on to the metadata-service, OIDC, Azure and Google strategies, and the first one that succeeds is used (e.g. az on an Azure host). The default chain would have refreshed and stopped at databricks-cli. The same behavior is now a Merge Danger bullet.
| require.NoError(t, err) | ||
|
|
||
| // WithOAuthArgument, WithTokenStore and WithoutRefresh. | ||
| assert.Len(t, receivedOpts, 3) |
There was a problem hiding this comment.
[nit] assert.Len(t, receivedOpts, 3) still passes if the noRefresh branch appended a different option, or if a refactor reordered the list, so the no-refresh behavior could disappear without this test failing. One way to test the behavior instead: pass receivedOpts plus a u2m.WithTokenStore holding an expired token to u2m.NewPersistentAuth, and assert that Token() returns u2m.ErrRefreshDisabled.
There was a problem hiding this comment.
Strengthened in 76d9ee3. The test now passes the options Configure returned, plus a u2m.WithTokenStore holding an expired token, to u2m.NewPersistentAuth, and asserts that Token() returns u2m.ErrRefreshDisabled. Replacing WithoutRefresh with another option in Configure makes it fail.
e503a62 to
76d9ee3
Compare
|
Thanks. Both questions are answered below; the inline nits are fixed in 76d9ee3. Question 1 (expired cached token uploads no event): I accept that. An install whose cached Question 2 (bounded synchronous PreRunE): I'd keep the background resolution. Bounding PreRunE with Outside-diff nit: updated. Evidence now says the tests pass locally on Linux and in CI on Linux, macOS and Windows, and the "Untested" bullet now says that only the Evidence timings were measured on Linux alone. |
71b4c1a to
29d7bbc
Compare
76d9ee3 to
7e03112
Compare
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 7e03112. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 2 open design questions.
This PR stops aitools install from blocking on an unreachable workspace. Telemetry auth now resolves in the background with a 3 s cap, never refreshes an OAuth token, and logs only at debug level. Every earlier review thread is addressed at this head, the new tests pass under the race detector, and CI is green on Linux, macOS and Windows. What's left is non-blocking: drop the PreRunE/RunE split, fix a test that calls t.Fatal on a background goroutine, close a pre-existing upload gap in the deprecated skills install alias, and three test-convention nits. There are also two follow-up questions on design trade-offs you've already accepted.
Questions
- You've accepted that an install whose cached databricks-cli access token has expired uploads no install event. Databricks OAuth access tokens are short-lived and
aitools installoften runs on its own, so the loss may skew install metrics toward PAT users and people who ran another CLI command recently. Do you have a rough sense of that share? Would you add a note under Merge Danger that metrics built on this event undercount those users, or is leaving it unquantified fine for now? - Your replies explain why you kept the background resolution rather than resolving in
Uploador bounding a synchronousPreRunEwith the SDK's timeouts. That reasoning is only in PR comments right now. Would you add a short note on the rejected alternatives to the description so it stays with the change, or is the current Merge Danger section enough?
| func withTelemetryAuth(cmd *cobra.Command) { | ||
| var pending *pendingAuth | ||
| cmd.PreRunE = func(cmd *cobra.Command, args []string) error { | ||
| pending = startTelemetryAuth(cmd, args) | ||
| return nil | ||
| } | ||
| runE := cmd.RunE | ||
| cmd.RunE = func(cmd *cobra.Command, args []string) error { | ||
| err := runE(cmd, args) | ||
| pending.apply(cmd) | ||
| return err | ||
| } |
There was a problem hiding this comment.
[nice to have] withTelemetryAuth starts the resolution in PreRunE, waits for it in the RunE wrapper, and passes pending between them through the closure. Cobra runs only ValidateRequiredFlags/ValidateFlagGroups between the two hooks, so the split gains no overlap. It does overwrite any PreRunE the command has: one added to install later would be silently dropped. It also creates the nil-pending case that TestTelemetryAuthRunEWithoutPreRun exists for. Suggest doing both in the wrapper (p := startTelemetryAuth(cmd, args); err := runE(cmd, args); p.apply(cmd); return err) and dropping the PreRunE assignment and that test.
There was a problem hiding this comment.
I'd rather leave the split in this PR. #7469, stacked on this one, builds on the wrapper's current shape: withTelemetryAuth returns a handle and the exit wait becomes opt-in, and it rewrites exactly these lines. Moving the start into the RunE wrapper here makes #7469 conflict on restack. The behavior is the same either way, since cobra runs only flag validation between the two hooks. This can be folded in on top of the stack in a follow-up.
There was a problem hiding this comment.
Our review found this too: three of our reviewers flagged the replaced PreRunE. One note: #7470 applies this helper to four more commands through instrumentCommand. So a PreRunE that someone later adds to update, uninstall, list or version is dropped with no error. If you keep the split, panic when cmd.PreRunE is already set, so that the conflict fails at once.
There was a problem hiding this comment.
Thanks. I tried the guard: withTelemetryAuth panicking when PreRun or PreRunE is already set. It conflicts with #7469, which rewrites these same lines to return the telemetryAuth handle, so I left it out of this PR. None of the commands that #7470 wraps (install, update, uninstall, list, version) has a PreRun or PreRunE today, so nothing is dropped now. The guard, or moving the start into the RunE wrapper, fits a follow-up on top of the stack, where instrumentCommand applies the helper to all five commands.
| } | ||
|
|
||
| func TestTelemetryAuthRunEWithoutPreRun(t *testing.T) { | ||
| setTelemetryAuth(t, time.Second, func(cmd *cobra.Command, args []string) error { t.Fatal("must not resolve"); return nil }) |
There was a problem hiding this comment.
[nit] This resolver calls t.Fatal, but resolvers only run on the background goroutine. If a change made it run, t.Fatal would fire off the test goroutine, probably after the test returned, since RunE does not wait when pending is nil. The test would then pass, or panic with "Fail in goroutine after TestTelemetryAuthRunEWithoutPreRun has completed", instead of failing cleanly. Record an atomic.Bool in the resolver and assert it on the test goroutine, as TestTelemetryAuthSkippedWhenTelemetryDisabled does with called. This goes away if the PreRunE split is removed.
There was a problem hiding this comment.
Fixed in d25f879. The resolver now records the call on a buffered channel, and the test asserts on the test goroutine (in a t.Cleanup) that nothing was recorded, so a stray call fails the test cleanly instead of calling t.Fatal off-goroutine.
| return | ||
| } | ||
| if p.cfg != nil { | ||
| cmd.SetContext(cmdctx.SetConfigUsed(cmd.Context(), p.cfg)) |
There was a problem hiding this comment.
[nit] This existed before this PR, but it shows here: apply puts the config on the context of the command it wraps. The deprecated experimental aitools skills install alias builds its own NewInstallCmd() and runs installCmd.Execute() (cmd/aitools/legacy_skills.go:45-58), while root uploads from the alias command's context. So the alias always ends with "no auth config available; skipping telemetry upload", and on an unreachable host it first waits up to 3 s for a config it then drops. A one-line fix in the alias: err := installCmd.Execute(); cmd.SetContext(installCmd.Context()); return err.
There was a problem hiding this comment.
Fixed in d25f879. After installCmd.Execute(), the alias now copies installCmd.Context() back onto its own command, so the config resolved for telemetry reaches root's upload. TestLegacySkillsInstallKeepsTelemetryAuthConfig covers it, and it fails without the fix.
| // It applies to that config only. A bundle's workspace client and the config | ||
| // of a profile picked interactively still refresh, so combine it with | ||
| // [SkipLoadBundle] and [SkipPrompt]. | ||
| func SkipTokenRefresh(ctx context.Context) context.Context { |
There was a problem hiding this comment.
[nit] SkipTokenRefresh/shouldSkipTokenRefresh have no test, while SkipPrompt and SkipLoadBundle each have one in cmd/root/auth_options_test.go. .agents/rules/testing.md asks to extend the test file when adding functionality. Add a TestSkipTokenRefresh that mirrors the two existing tests.
There was a problem hiding this comment.
Added TestSkipTokenRefresh in cmd/root/auth_options_test.go in d25f879, mirroring the two existing tests.
| write_token_cache expired-access-token 2000-01-01T00:00:00Z | ||
| trace $CLI aitools install --skills-only --scope=global --agents=claude-code --profile test-profile | ||
| trace print_requests.py //oidc/v1/token //telemetry-ext | ||
| trace jq -r '.tokens["test-profile"].access_token' "$HOME/.databricks/token-cache.json" |
There was a problem hiding this comment.
[nit] .agents/rules/testing.md prefers gron.py | grep <field> over inline jq paths for single-value lookups, because the gron line names the path in output.txt. Suggest trace gron.py "$HOME/.databricks/token-cache.json" | grep access_token, then regenerate the output with -update.
There was a problem hiding this comment.
Switched to trace gron.py "$HOME/.databricks/token-cache.json" | grep access_token and regenerated the output in d25f879.
| @@ -0,0 +1,217 @@ | |||
| package aitools | |||
There was a problem hiding this comment.
[nit] The tests for the new code in cmd/aitools/telemetry.go live in a new telemetry_auth_test.go with no matching source file. .agents/rules/testing.md asks for one test file per source file, extended as the source grows. Either move the auth code (telemetryAuthBudget through debugOnlyHandler) into cmd/aitools/telemetry_auth.go, which keeps telemetry.go about building events, or move these tests into telemetry_test.go.
There was a problem hiding this comment.
I'd rather not move them in this PR. #7469 and #7470, stacked on this one, extend both the auth code in telemetry.go and telemetry_auth_test.go, so moving either here makes both PRs conflict on restack. It's a pure move, so it fits better as a follow-up on top of the stack (e.g. the auth code into telemetry_auth.go, keeping telemetry.go about building events).
7e03112 to
d25f879
Compare
|
Thanks. Answers to the two questions are below; the inline threads have replies, with fixes in d25f879. Question 1: I'm fine leaving it unquantified. Merge Danger already says that an install whose cached Question 2: added. The description now has a short paragraph on the rejected alternatives (resolving in |
29d7bbc to
2c02b39
Compare
d25f879 to
fa4413b
Compare
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Approval from simonfaltum's review agent at fa4413b, on Simon's decision.
The design questions from the earlier automated reviews no longer block this PR: your answers are accepted, and any still-open ones are non-blocking. The changes since the last automated review (7e03112 → fa4413b) address its comments.
rugpanov
left a comment
There was a problem hiding this comment.
Also found by our reviewers, already asked in simonfaltum's review (Question 1): an install whose cached databricks-cli token has expired uploads no install event. Our note: our devil's-advocate reviewer built the alternative that passes the unresolved config to Upload (about 22 lines). It reported the same exit time as this PR with the PR's own repro (3.37 s and 3.38 s against a host that refuses the connection), and it keeps the install event when the cached token has expired.
| # Isolate HOME so the real machine's agents and skills are not seen. | ||
| sethome home | ||
| # Agent detection needs ~/.claude; prefer USERPROFILE on Windows. | ||
| mkdir -p "${USERPROFILE:-$HOME}/.claude" |
There was a problem hiding this comment.
[nit] .agents/rules/testing.md says: "Commit static test inputs into the acceptance test directory; do not create them in script at test time." #7467 fixed the same pattern in apps-init-tip with a committed home/.claude/.gitkeep. This script and acceptance/cmd/aitools/telemetry/install/script:4 still run mkdir -p "${USERPROFILE:-$HOME}/.claude". Commit home/.claude/.gitkeep in both directories and remove the mkdir.
There was a problem hiding this comment.
Done in 03fa288. Both install-host-metadata and telemetry/install now commit home/.claude/.gitkeep, and the mkdir is gone from their scripts.
| title "--debug still shows the hostmetadata line, at debug level" | ||
| # A fresh cache, so the first run's negative-cache entry does not skip the fetch. | ||
| export DATABRICKS_CACHE_DIR="$PWD/debug-cache" | ||
| $CLI aitools install --skills-only --scope=global --agents=claude-code --debug 2>&1 | grep -o 'Debug: \[hostmetadata\] failed to fetch host metadata for [^,]*' |
There was a problem hiding this comment.
[nit] .agents/rules/testing.md asks to use the acceptance/bin/ helpers before inline grep or jq pipelines. This line pipes the --debug run through grep -o without trace. Use trace $CLI ... --debug &> LOG.debug, and then contains.py 'failed to fetch host metadata' < LOG.debug.
There was a problem hiding this comment.
Done in 03fa288. The --debug run is now traced into LOG.debug, and contains.py checks it for Debug: [hostmetadata] failed to fetch host metadata for https://unreachable.test and for no Warn: [hostmetadata] line. I redirect its stdout to /dev/null because contains.py echoes its input, and the full debug log has timestamps and a pid. If the line is missing or still at Warn, the test fails with Exit code: 1; I checked this by inverting the assertion.
19f3090 to
96084b5
Compare
0b368c6 to
03fa288
Compare
|
Re the review summary (expired cached token): see my answer to Question 1 in simonfaltum's review (#7468 (comment)); Merge Danger states the trade-off. Passing the unresolved config to |
`aitools install` resolved auth for telemetry in PreRunE, before doing any work, so an unreachable workspace blocked the install for as long as the SDK retried. It now resolves auth in the background while the install runs and waits for it at most 3 s after the command started. Nothing is resolved when `DATABRICKS_CLI_DISABLE_TELEMETRY` is set (new `telemetry.Disabled`), and a run that logged no event does not wait. ```diff aitools install - PreRunE: tryConfigureAuth - MustAnyClient(cmd) # blocks while the SDK retries host metadata - RunE: install skills + PreRunE: startTelemetryAuth # returns at once + if telemetry.Disabled → skip + go MustAnyClient(shadow cmd) # shares cmd's flags (--profile), not its context + # SkipTokenRefresh: never refreshes an OAuth token + RunE wrapper + runE: install skills # runs while auth resolves + pending.apply(cmd) + no event logged → return + wait until start + 3 s, at most + succeeded → SetConfigUsed(cmd ctx) + failed or not finished → no config, no upload root: telemetry.Upload ``` The background resolution never refreshes an OAuth token. The process may exit before it finishes, and the server rotates the single-use refresh token, so an abandoned refresh could lose the new one and log the user out. The new `root.SkipTokenRefresh` makes `MustWorkspaceClient` and `MustAccountClient` use `auth.CredentialsWithoutRefresh`, whose databricks-cli strategy passes the new `u2m.WithoutRefresh`: the cached token is used only while it is valid, and an expired one fails auth, so nothing is uploaded. The shadow command logs through `debugOnlyHandler`, so a background warning such as `[hostmetadata] failed to fetch host metadata ...` shows only with `--debug` and does not interrupt the install's output or prompts. This relies on root's host-metadata resolver logging through the command's logger (#7390). A panic in the background resolution is recovered and logged at debug level; the install does not fail. Changelog fragment: `.nextchanges/cli/aitools-install-telemetry-auth.md`. Co-authored-by: Isaac <no-reply@databricks.com>
03fa288 to
d13de92
Compare
Stacked on databricks#7468. Stacked on: aitools: bound best-effort telemetry auth ## Summary Every `databricks aitools install` run now logs an `aitools_command_event` funnel next to the existing `aitools_install_event`, so the telemetry shows where runs stop: ```text install RunE STARTED (every run past flag parsing) validate flags, --agents names --> FINISHED FAILURE INVALID_FLAGS | UNKNOWN_AGENT STEP FLAGS_VALIDATED --path dump --> STEP SKILLS_DUMPED, FINISHED SUCCESS (0 skills written: FINISHED NOTHING_TO_DO) scope prompt --> FINISHED CANCELLED USER_ABORTED | FAILURE PROMPT_FAILED STEP SCOPE_RESOLVED picker --> FINISHED CANCELLED (nothing checked) no agent detected: FINISHED NOTHING_TO_DO STEP AGENTS_SELECTED confirm prompt --> FINISHED CANCELLED (declined) STEP PLAN_CONFIRMED executePlan STEP SKILLS_INSTALLED (skills group installed) STEP PLUGINS_INSTALLED (at least one plugin installed) FINISHED SUCCESS | FAILURE <category> ``` Each event carries `command`, `stage`, `interactive`, `install_mode` (DEFAULT / SKILLS_ONLY / PATH) and, once known, `scope`. FINISHED also carries `outcome`, `error_category` and the targeted `agents`. Every event of a run shares the entry's `cmd_exec_id`. The CLI sets no calling-agent field on these events: the upload's User-Agent already has `agent/<name>`, the calling agent's Databricks tooling (`aitools-agent/...`) and `cmd-exec-id`. `error_category` comes from the existing `classifyInstallError`. Failures in flag validation, agent lookup and prompts are tagged with a category where they happen (a prompt's own failure is PROMPT_FAILED whatever its type, except a Ctrl-C abort; an error before the prompt opens keeps its own category). `Error()` / `Unwrap()` pass through, so the user still sees the same message: ```diff classifyInstallError(err) *installer.BlockedError / *installer.SkillError -> their category + *categorizedError (withCategory, promptError) -> the tagged category stdlibErrorCategory(err) ``` Some runs get an explicit outcome instead of the one their return value implies: - Returns nil but installed nothing (e.g. the only detected agent's plugin install failed, which is a warning): FAILURE with the first failing agent's category. This includes a non-interactive run whose detected agents are all skipped for the scope (UNSUPPORTED_SCOPE). The interactive picker never offers skipped agents, so the same machine finishes NOTHING_TO_DO there. - Fails only because a named agent failed or was skipped (`--agents`): the category comes from that agent's outcome (e.g. UNSUPPORTED_SCOPE), not UNCATEGORIZED. - `--output json` failure: classifies the real error, not the `ErrAlreadyPrinted` sentinel. - `--project --global` and `--agents ","`: INVALID_FLAGS, not PROMPT_FAILED / UNKNOWN_AGENT. Runs that end before PLAN_CONFIRMED (a flag error, a cancelled prompt, no agent detected, a `--path` dump) upload their events only if the background telemetry auth has already resolved; they never wait for it. Runs past PLAN_CONFIRMED keep the bounded wait. An unknown `--agents` name now fails before the scope prompt instead of after it (changelog fragment `.nextchanges/cli/aitools-install-agents-before-scope-prompt.md`). Otherwise no change to output, prompts, exit codes or error text. ## Evidence - **Before** (non-test code of this PR reverted, test files kept): acceptance `cmd/aitools/telemetry/install` fails; the upload has only `aitools_install_event`: ```text --- FAIL: TestAccept/cmd/aitools/telemetry/install -{"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}} -{"aitools_command_event":{..."stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED"...}} -... SCOPE_RESOLVED, AGENTS_SELECTED, PLAN_CONFIRMED, SKILLS_INSTALLED ... -{"aitools_command_event":{..."stage":"FINISHED","outcome":"SUCCESS",...,"agents":["CLAUDE_CODE"],"scope":"GLOBAL"}} ``` **After:** the same test passes. `install-failure` pins the failure paths past PLAN_CONFIRMED: missing skill, manifest 404 / 500, named agent skipped for the scope. Runs that end earlier upload only if auth already resolved, which depends on timing, so they are covered by unit tests instead. - Unit tests in `cmd/aitools/funnel_test.go` (for the funnel code in `funnel.go`) cover every early return and each success path (e.g. `TestInstallFunnelEarlyExits`, `TestInstallFunnelAbortedScopePromptKeepsNamedAgents`, `TestInstallFunnelPathDumpSuccess`, `TestInstallFunnelPathDumpWithNoSkillToWrite`, `TestInstallFunnelPluginSuccess`, `TestInstallFunnelInteractiveConfirm`, `TestInstallFunnelDetectedPluginFailureIsNotSuccess`, `TestInstallFunnelSkillsFailureThroughJSON`); they and `TestAitoolsEventsWireKeys` pass with `-count=1`. - `TestInstallUnknownAgentFailsBeforeScopePrompt`: with prompts supported, `--agents bogus` fails without showing the scope prompt (fails if `resolveAgentNames` moves back below the prompt). - The upload wait: `TestInstallEarlyExitDoesNotWaitForTelemetryAuth` (a flag error returns at once while auth is still resolving; fails after 10 s with the wait put back) and `TestInstallPastPlanConfirmedWaitsForTelemetryAuth` (auth that finishes only after the install step still reaches the upload; fails without the wait). - `go build ./...`, `./task test-exp-aitools`, `go test ./libs/telemetry/... ./cmd/root/...`, `./task lint-q`, `./task fmt-q`, `./task checks`, and `go vet` with `GOOS=windows` / `GOOS=darwin` pass. ## Merge Danger **Door:** two-way Telemetry only. Reverting removes the events; nothing on disk or in the install state changes. **Blast Radius:** telemetry - More events per `aitools install` upload (about eight instead of one), sent at exit and only when auth resolved. Runs that stop before the plan is confirmed (flag errors, a cancelled prompt, no agent detected) and `--path` dumps logged no event before, so they made no telemetry request. They now upload only if the background auth has already resolved when they end, and never wait for it, so an unreachable or slow-to-authenticate workspace adds nothing to them. When auth has resolved, the upload itself can still take up to its timeout (`DATABRICKS_CLI_TELEMETRY_TIMEOUT`, default 3 s) against a slow workspace. The flip side: a run that ends in milliseconds (e.g. a flag error) usually ends before auth resolves, so its events are often dropped; a run that stopped at a prompt usually had time to resolve auth. The funnel therefore under-counts the fastest early exits, and their STARTED events, relative to slower ones; the code comment next to the rule says the same. Runs past the plan confirmation keep the bounded auth wait. - Not covered: the funnel trusts the skills installer, which reports no per-agent delivery. So two cases still finish SUCCESS with SKILLS_INSTALLED although an agent got no skill: every skill was filtered out (experimental, or below the minimum CLI version; databricks#7481, further up this stack, makes the installer report this and the funnel finish NOTHING_TO_DO), or the installer could not write an agent's destination (it warns and continues). `--output json` and `aitools_install_event` report these runs as installed today too. The `--path` dump knows how many skills it wrote, so a dump that writes none finishes NOTHING_TO_DO. A run killed by Ctrl-C outside a prompt, or rejected by flag parsing, uploads no funnel events. - Not tested here: interactive prompts on Windows (the funnel paths are covered by unit tests with mocked prompts and compile-checked for Windows). This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…atabricks#7470) Stacked on databricks#7469. Stacked on: aitools: add install funnel telemetry ## Summary `aitools update`, `list`, `version` and `uninstall` now log the same `aitools_command_event` funnel as `aitools install` (STARTED, then FINISHED with the outcome and error category) and resolve auth best-effort in the background so the events can upload at exit. `list` and `version` never wait for that resolution at exit, and neither do `update` and `uninstall` runs that end before their work (flag and scope errors, a declined confirm, nothing installed in the scope, a `--skills` name that is not installed), the same rule databricks#7469 applies to `install` before its plan is confirmed; their events upload only if auth has already resolved. ```diff NewUpdateCmd / NewListCmd / NewVersionCmd / NewUninstallCmd - return cmd + return instrumentCommand(cmd, protos.AitoolsCommand<X>) + cmd.RunE = withCommandFunnel(...) // STARTED; FINISHED from RunE's error + withTelemetryAuth(cmd) // background auth; no wait at exit unless + // RunE calls waitForTelemetryAuth (update and + // uninstall, once a scope has state) ``` ```text finish(err) nil -> SUCCESS errors.Is(err, ErrNothingInstalled) -> NOTHING_TO_DO (still exits 1 with the same text) prompt aborted / no agents picked -> CANCELLED otherwise -> FAILURE, classifyInstallError(err) (scope flag errors tagged INVALID_FLAGS, as in install) RunE returns nil, but uninstall confirm answered "no" -> CANCELLED uninstall removed nothing and nothing failed -> NOTHING_TO_DO every plugin/skill update or removal attempted -> FAILURE with the first error's category failed and nothing else succeeded (still exits 0) (each via funnelFrom(ctx).finishWith) ``` Installer additions: `ErrNothingInstalled` (same error text as before), `PluginUpdate.Err` (failed plugins are returned after the updated ones; `update` prints only the updated ones), and `UpdateResult.CheckErr` (the manifest fetch error a `--check` run only warns about), and `UninstallSkillsWithResult` returning `UninstallResult{PluginsRemoved, SkillsRemoved, PluginFailures, SkillFailures}`, which replaces `UninstallSkills` and `UninstallSkillsOpts`. A plugin record for an agent this CLI does not know counts as a failed update or removal (`ErrUnknownAgent`, UNKNOWN_AGENT), and a pre-state (legacy) install returns `ErrLegacyInstall` (same text, STATE_INVALID). `uninstall --skills` with a skill that is not installed returns a `SkillError` (same text). Full uninstall now removes skills in sorted order. Changelog: `.nextchanges/cli/aitools-telemetry-funnel.md`. ## Evidence - **Before** (`return instrumentCommand(cmd, protos.AitoolsCommandList)` in `list.go` reverted to `return cmd`): ```text --- FAIL: TestInstrumentedCommandsResolveAuth Expected value not to be nil (list has no PreRunE) --- FAIL: TestListFunnelFailure expected [STARTED FINISHED:FAILURE:NETWORK], actual [] ``` **After** (current head): ```text ok cmd/aitools TestInstrumentedCommandsResolveAuth, TestListFunnelFailure, TestUninstallFunnelWhenRemovalsFail, TestUpdateFunnelWhenPluginsFail, TestNothingInstalledFinishesNothingToDo, TestInstrumentedCommandsDoNotWaitForTelemetryAuth, TestUpdateAndUninstallWorkWaitsForTelemetryAuth ``` - **No wait**: with `instrumentCommand` forced to wait, `TestInstrumentedCommandsDoNotWaitForTelemetryAuth` fails for version, list, the update/uninstall flag errors and update/uninstall with nothing installed (10 s each, budget set to 10 s); with `waitForTelemetryAuth` a no-op, `TestUpdateAndUninstallWorkWaitsForTelemetryAuth` fails for both commands. With `PreRunE` cleared in `instrumentCommand`, `TestInstrumentedCommandsResolveAuth` fails for update, list, version and uninstall. - Built CLI, fake token, `DATABRICKS_HOST=https://127.0.0.1:1` (refused): `aitools version` 18 ms, `update --scope=bogus` 17 ms, `update` and `uninstall` with nothing installed 17-18 ms, `uninstall` with one skill recorded 3.0 s. - Acceptance `cmd/aitools/telemetry/update-uninstall` pins the uploaded STARTED/FINISHED events of `update` and `uninstall` after an install. `list`, `version`, and `update`/`uninstall` with nothing installed upload only if auth has already resolved, which depends on timing, so their events are pinned by unit tests instead. `workspace-unreachable` checks that the background resolution prints nothing on install, update and uninstall runs that wait for it (with the handler's level clamp removed, it shows the hostmetadata warning). - At this head: `go build ./...`, `./task test-exp-aitools`, `cmd/aitools`, `libs/aitools` and `libs/telemetry` unit tests, the `cmd/aitools` acceptance tests, `./task lint-q`, `./task fmt-q`, `./task checks`, and `go vet` for windows and darwin: all pass. ## Merge Danger **Door:** two-way **Blast Radius:** latency - `update` and `uninstall` runs on a scope with something installed now contact the configured workspace: up to about 6 s longer on a slow workspace (3 s auth wait + 3 s upload), at least 3 s per run while connections are refused or time out. - `list`, `version`, and `update`/`uninstall` runs that end at a flag or scope error, a declined confirm, with nothing installed or at a `--skills` name that is not installed never wait for auth. When the resolution has already finished (e.g. a PAT or a valid cached token, with host metadata cached), they still upload and can take up to 3 s longer (the upload timeout) on a slow workspace; otherwise their events are dropped. - Exit codes and output are unchanged; uninstall and update failures that were only warnings still exit 0 and are reported only in telemetry. - The background resolution never refreshes an OAuth token (databricks#7468); with an expired cached token the events of these commands are dropped, as for `install`. - Not tested here: Windows and macOS (run on Linux only); a real workspace upload. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
## Release v1.27.0 ### Notable Changes * `databricks aitools install`, `update` and `uninstall` now exit with code 1 when they do nothing they were asked to. `install` fails when it sets up nothing: no coding agent is detected, every agent is skipped or fails to install, or no skill in the release can be installed (also with `--path`). `update` fails when every update or check it attempts fails, and `uninstall` when every removal it attempts fails. Scripts that ran these commands as a best-effort step and relied on exit code 0 should handle the new exit code. ([#7481](#7481)) * Add `file:<path>` selectors to `pipelines run` and pipeline `bundle run`. `--refresh` and `--full-refresh` now use space-separated lists, such as `--refresh "orders payments"`, instead of comma-separated lists. Single names containing spaces now require backticks, e.g. ``--refresh 'table:`order details`'``. ([#7184](#7184)) * The Databricks CLI is now available on PyPI as `databricks`: run it with `uvx databricks`, or install it with `uv tool install databricks`, `pipx install databricks` or `pip install databricks`. The package contains only the prebuilt CLI binary. Uninstall the legacy `databricks-cli` package first, since both install a `databricks` command. ([#7704](#7704), [#7777](#7777)) ### CLI * Add `databricks sandbox fs` commands (`cp`, `ls`, `cat`, `mkdir`, `rm`) to transfer files between your local filesystem and sandboxes using `<sandbox-id>:/path`. ([#7372](#7372)) * `databricks fs` commands retry reads (stat, list, open) and deletes on DBFS and Volumes when the server returns a transient error or does not respond within the HTTP timeout. ([#7556](#7556)) * Retry UC Volumes file operations (`fs cp`, `fs ls`, `fs rm`, bundle uploads to Volumes) when the Files API returns HTTP 408, 429, 502, 503 or 504 (for example "AWS S3 is throttling requests"), or on a transient network error; this regressed in v1.10.0. ([#7559](#7559)) * Stop warning when the host metadata request (`/.well-known/databricks-config`) times out; the CLI already falls back to your config, and the message is still shown with `--debug`. ([#7752](#7752)) * Add `databricks aitools auto-install disable` and `enable` to turn off (or back on) the automatic installation of Databricks AI tools on CLI upgrades. A global-scope `databricks aitools uninstall` also turns it off. ([#7701](#7701)) * Stop printing the Databricks AI tooling install hint on `auth`, `configure`, `current-user me`, `version`, `help`, and shell-completion commands. ([#7452](#7452)) * When Codex, Cursor, GitHub Copilot CLI, OpenCode, Gemini CLI, Antigravity, Pi, Goose or Kiro run the CLI without Databricks skills installed for that agent, the CLI now prints a one-line hint to install them with `databricks aitools install --agents <agent> --scope global`, previously shown only to Claude Code. ([#7586](#7586)) * `databricks aitools install` now rejects an unknown `--agents` name before it shows the scope prompt, instead of after. ([#7469](#7469)) * `databricks aitools install` no longer asks "Proceed?" after the agent picker; the picker selection confirms the plan. The picker title now shows the install scope, its hint says Enter installs, `--skills-only` runs label plugin agents "skills" and pre-check every agent whose config dir exists, and the plan summary after it is reduced to the scope line. ([#7478](#7478)) * `databricks aitools install --output json` reports more specific `error_category` values: `SKILLS_FETCH_FAILED`, `RELEASE_LOOKUP_FAILED`, `RATE_LIMITED`, `MANIFEST_INVALID`, `STATE_INVALID` and `PLUGIN_CLI_TIMEOUT`. A coding agent CLI that does not answer in time is now reported as `timed out after <duration>`, followed by any stderr it wrote before it was stopped. ([#7475](#7475)) * `databricks aitools install` no longer prompts when a coding agent runs it, even in a pseudo-terminal; it uses the non-interactive defaults instead and prints a note saying so. ([#7477](#7477)) * `databricks aitools install` now prints the directory it installed skills into (for example `Installed 1 skill in ~/.databricks/aitools/skills.`), so agents can open the installed `SKILL.md` files without searching for them. ([#7668](#7668)) * `databricks aitools install` now ends a failed install with a step to recover, in the text output and in a new `recovery` field of `--output json`. ([#7480](#7480)) * `databricks aitools install` now installs plugins for the other coding agents even when installing skill files fails. ([#7479](#7479)) * `databricks aitools install` now installs skill files for a detected coding agent whose CLI is not on PATH (for example a Claude Code desktop or IDE install), unless it already has the plugin, says how to switch to the plugin later, and warns when the current project also has the plugin. `databricks aitools update` no longer tries to install the plugin for such an agent, and no longer writes skill files next to a plugin that such an agent already has. ([#7482](#7482)) * Fix `databricks aitools install` waiting a long time on an unreachable or slow workspace while it resolves authentication for telemetry; that wait is now limited to a few seconds. ([#7468](#7468)) * `databricks aitools install`, `update` and `list` now refuse a skills manifest that names a skill or file outside the skills directory, before writing any skill files. A skill whose name or file paths are not portable is skipped with a warning, and `update` keeps its installed copy. ([#7722](#7722)) * `databricks aitools install` and `databricks aitools update` refresh only the marketplace the Databricks plugin is listed in, so another unreachable marketplace no longer blocks them. When that refresh fails, both commands warn, install or update the plugin from the local copy of the marketplace, and record the plugin version as unknown. ([#7473](#7473)) * `databricks aitools install` and `databricks aitools update` now wait up to 20 seconds instead of 5 for a coding agent's CLI to answer the plugin capability check, so a slow first start is less likely to report the agent as unavailable. ([#7472](#7472)) * `databricks aitools install` no longer fails a plugin-only install when the lookup of the latest skills release on GitHub fails or is rate-limited. The plugin is installed and its version is recorded as unknown. ([#7474](#7474)) * `databricks aitools` commands now send usage telemetry, and the CLI records when it recommends installing Databricks AI tools. When the configured workspace is slow or unreachable, `aitools install`, `update` and `uninstall` can take up to about 6 seconds longer, and `list`, `version` and runs that stop early up to about 3 seconds. Set `DATABRICKS_CLI_DISABLE_TELEMETRY=1` to opt out. ([#7467](#7467), [#7469](#7469), [#7470](#7470)) * `databricks aitools install` and `databricks aitools update` now report a coding agent CLI shim (`.cmd` or `.bat`) on Windows that prints its plugin help and then exits with an error as `shim-exited-with-error`, with the command to run to see the error, instead of reporting the CLI as not on PATH. ([#7476](#7476)) * `apps init` no longer suggests installing skills when every detected coding agent already has the Databricks plugin. ([#7467](#7467)) * `databricks auth login` now gives up on the optional workspace lookup after about 5 seconds instead of retrying for about 5 minutes, and explains the fallback to the manual workspace ID prompt. ([#7522](#7522)) * Throttle Claude Code's native plugin install hint separately from the text hint: it prints at most once per 10 minutes per session, so the install dialog still comes up when a subagent, a hook, or a command outside a Claude Code subprocess runs the session's first `databricks` command. ([#7692](#7692)) * Print Claude Code's native plugin install hint alongside the text hint, so Claude Code offers a one-click install of the Databricks plugin. Claude Code now counts as having the tooling only when it loads it: skills only in `~/.databricks/aitools/skills` don't count, and a project- or local-scope plugin counts only when the CLI runs inside that project. `auth token`, `auth docker token` and CLI subprocesses no longer print install hints. ([#7585](#7585)) * Interactive prompts and pagers now work in Git for Windows' bash and other MSYS2 shells when they run under ConPTY (the default since Git for Windows 2.47) or directly in a Windows console. ([#7699](#7699)) * Add the `databricks feedback` command to send feedback about the CLI to Databricks. ([#7724](#7724)) * Fix shell completion in `databricks fs` commands for local paths typed with a leading `./` or repeated separators, which previously offered no suggestions. ([#7679](#7679)) * Fix a panic in `databricks labs` commands when a non-string SDK setting such as `DATABRICKS_DEBUG_HEADERS` or `DATABRICKS_RATE_LIMIT` is set in the environment. ([#7673](#7673)) * Fix a panic in `databricks labs install` for projects whose `labs.yml` has no `install` section. ([#7673](#7673)) * `NO_COLOR` no longer disables prompts, spinners or the list pager; they run without color. ([#7700](#7700)) * Add the Beta `databricks pipelines test` command, which runs the tests of a pipeline, and the `testing` property of pipelines in bundles. ([#7429](#7429)) * Add `databricks sandbox forward`, which serves an app running in a sandbox at a local URL. ([#7395](#7395)) * Bound the whole telemetry upload at exit, including config resolution and authentication, by its timeout, so a stalled auth or telemetry endpoint no longer keeps a command from exiting. ([#7467](#7467)) * `databricks version --check` now suggests `databricks aitools install` on stderr when a coding agent is detected without the Databricks plugin or skills. ([#7471](#7471)) ### AI Runtime * `air get -o json` now reports `PENDING` while the run waits for compute, matching `air list`, and includes the run's `status_message`. ([#7455](#7455)) * Wrap long failure reasons and log lines in the `air list` info and logs panes, keep the MLflow link (labeled with its retry) when a run's last retry failed before starting, and point `air logs` at earlier retries when the last one has no logs. ([#7456](#7456)) * `databricks air logs --node` now reports an out-of-range node index when it can determine the run's node count, instead of looking like a successful run with no logs. ([#7516](#7516)) * Add experiment, duration, accelerator, and dashboard metadata to `air list --output json`. For retried runs, `started_at` now reports the latest attempt, matching `air get`. ([#7568](#7568)) * `air list` shortens MLflow run names and unnamed-run IDs by keeping their end behind a leading `…`, and its columns no longer overflow on wide (for example CJK) text; `air get` shows an unnamed run's full MLflow run ID. ([#7582](#7582)) * Shrink the Experiment and User columns of the interactive `air list` table to fit the terminal width, keeping the end of a shortened experiment name. ([#7583](#7583)) * Show the experiment, accelerators and configuration in the `air list` info pane for runs where it showed `N/A`. ([#7587](#7587)) * Strip terminal control characters from the MLflow run, experiment and user names shown by `air list` and `air get`, so a run name cannot clear or retitle the terminal of whoever lists it. ([#7601](#7601)) * `air logs`, `air get`, and `air run --watch` no longer make an extra credential-check request before fetching the run, and `air get` reuses the configured workspace ID instead of looking it up. ([#7613](#7613)) * `air list` no longer waits on extra current-user requests before listing runs, which makes the default list about 150ms faster. ([#7615](#7615)) * `air list --all-status` requests larger index pages, so it makes far fewer round trips once the server supports `page_size`. ([#7616](#7616)) * Fit the interactive `air list` table to narrow terminals: when the Experiment and User columns are at their floors, shorten Started to `MM-DD HH:MM` and size the MLflow column to its labels, then hide Accelerators, User, MLflow and Progress in that order, naming the hidden columns in the hint line. ([#7629](#7629)) * `air get -o json` now includes `mlflow_run_id` and `mlflow_experiment_id`, so a run can be handed to MLflow-based steps without parsing `mlflow_url`. They identify the MLflow run `mlflow_url` links to, including one from an earlier retry that `mlflow_retry` names. ([#7636](#7636)) * `air get` and the `air list` info pane now show the environment of runs on the Databricks AI environment (`databricks_ai_v*`) instead of `N/A`. ([#7637](#7637)) * `air run` now rejects a `code_source.include_paths` path that selects no files instead of uploading a snapshot without it; glob patterns and pathspec magic are left to git. ([#7643](#7643)) * `air convert-to-dabs` now suggests the `--output-dir` that converts a working-tree workload whose `code_source.root_path` is outside the bundle directory, instead of suggesting `sync.paths`. ([#7644](#7644)) * `air run` now lists a `code_source.root_path` outside a git repository relative to `root_path`, so running from a subdirectory of `root_path` no longer uploads an empty or partial snapshot. ([#7646](#7646)) * `air list --all-status` now fills `--limit` past purged or inaccessible indexed runs. ([#7743](#7743)) * `air run` now warns about missing PyTorch when a pinned snapshot's `include_paths` selects a Python file. ([#7744](#7744)) * `air cancel JOB_RUN_ID` now rejects run IDs that are not AI Runtime runs instead of cancelling unrelated Jobs runs. In `-o json` output, the `failed[].error` text for a run that does not exist or that you cannot view now starts with `failed to get run <id>:` instead of the cancel API error. ([#7569](#7569)) * `air get -o json` now reports `mlflow_retry` when the MLflow link comes from an earlier retry because the run's last retry never reached MLflow. ([#7620](#7620)) * `databricks air run --help` now shows how to install the databricks-ai-runtime coding agent skill. ([#7652](#7652)) * `air run --watch -o json` now reports each lifecycle `STATUS` transition once and never reports a non-terminal state after the run has ended, including when logs are served from MLflow artifacts. ([#7621](#7621)) * `air convert-to-dabs` now explains the `--output-dir` workaround when `code_source.root_path` resolves to the bundle root, and notes when the generated job carries the default `max_retries`. ([#7579](#7579)) * `air run` now refuses to upload a `code_source` snapshot that selects no files (for example when `root_path` is git-ignored) and explains the cause, and `--dry-run` runs the snapshot file selection and reports the file count. ([#7570](#7570)) * Repeat the Jobs run link after live log streaming ends. ([#7571](#7571)) * `air get -o json` includes run details including flat `accelerator_type` and `num_accelerators`. Use `--include-config` to include stored config, which may contain plaintext `env_variables` and parameters; downloads over 1 MiB are omitted, and config-dependent `max_retries` is otherwise null. ([#7624](#7624)) * `air get` and the `air list` info pane no longer show `Max Retries 0` when the Jobs API leaves the task's retry limit out. For a run submitted with `air run`, the limit now comes from the run's uploaded config, shown as `3 (CLI default)` when the config doesn't set `max_retries`; it shows `N/A` when the limit can't be known (job runs, multi-container runs, legacy runs with no explicit value, or an unreadable config). ([#7623](#7623)) * Reorganize `databricks air` help: subcommands are grouped by task, `air run` has examples, and YAML fields are listed under headings, with pool fields marked optional and `command`, `compute.num_accelerators`, and `compute.accelerator_type` correctly marked required. ([#7512](#7512)) * `air run -h config` now lists the environment variables AI Runtime sets on every node (such as `MLFLOW_RUN_ID` and `CODE_SOURCE_PATH`), and `air logs --help` notes that `--download-to` cannot be combined with `--tail` or `--minutes`. ([#7655](#7655)) * Support full workspace accelerator types in `air list` filters while preserving family matching, and reject invalid filter values. ([#7515](#7515)) * `databricks air list --all-status` now warns when the AI Runtime run index is unavailable and it falls back to scanning Jobs runs. That fallback scan is capped at 250 runs, and a stuck index request fails after 15 seconds instead of retrying for minutes. `--all-status --filter user=<your email>` now uses the run index instead of scanning the workspace's full run history. ([#7619](#7619)) * Return structured JSON errors for invalid `air list --filter` values when `-o json` is used. ([#7612](#7612)) * `air list --all-status` no longer fails when a listed run has been purged. ([#7523](#7523)) * **Behavior change:** `databricks air logs` now prints the logs available now and exits, also for an active run, matching `kubectl logs`, `docker logs` and `databricks apps logs`. Pass `--follow` (`-f`) to stream new lines until the run completes; `--tail N --follow` prints the last N lines first. `databricks air run --watch` still submits and streams. ([#7658](#7658)) * `databricks air logs -o json` and `databricks air run --watch -o json` now emit a `WAITING` event, for example `{"type":"WAITING","message":"Waiting for accelerator compute capacity..."}`, when the reported waiting status changes. ([#7695](#7695)) * `databricks air logs` on a run that ended in the last two minutes now waits for the log index to settle before printing, instead of sometimes dropping lines that were indexed late. ([#7663](#7663)) * `air logs` and `air run --watch` now print the waiting status (for example `Waiting for accelerator compute capacity...`) as a line on stderr when stderr is not a terminal, instead of printing nothing until the run produces logs. ([#7657](#7657)) * Route AIR log links to the MLflow Logs page instead of the hidden artifact path. ([#7491](#7491)) * Invalid `compute.accelerator_type` errors and `air run -h config.compute` no longer list the CLI's built-in accelerator types, which could differ from the workspace's. ([#7520](#7520)) * `databricks air --help` now notes that GPU pools are not required to run workloads. ([#7581](#7581)) * `air run` now prints the job run URL with the `?o=<workspace-id>` suffix, matching `air get` and `air list`; `dashboard_url` in `air run -o json` and the `--watch` events gains it too. When `air list` cannot resolve the workspace ID, its links now omit `?o=` instead of ending in `?o=0`. ([#7573](#7573)) * `databricks air run` now reports an unknown workload YAML key by its field path, suggests the closest valid field, and points to `-h config` instead of naming an internal Go type. ([#7574](#7574)) * `air run` (including `--dry-run`) now warns before submitting when the `code_source` imports `torch` but PyTorch is not declared: the environment is the Standard environment (version 5 or later) with no `torch` dependency, so the run may fail after it waits for a GPU. ([#7656](#7656)) ### Bundles * Add `isolation_mode` to the `catalogs` resource, so a catalog can be deployed as `OPEN` or `ISOLATED`. ([#7463](#7463)) * `bundle validate` no longer warns when resource permissions and top-level or target permissions name the same principal. ([#7291](#7291)) * Retry bundle file uploads, reads and deletes when the server does not respond within the HTTP timeout ("request timed out after ... of inactivity"). ([#7521](#7521)) * Show how to recover when `bundle deploy` fails because an app's service principal does not exist or was deleted. ([#7533](#7533)) * Retry transient server errors when creating Lakebase (`postgres_*`) resources in the direct deployment engine. ([#7557](#7557)) * Fix `bundle plan` reporting the `spark_conf` and `custom_tags` the backend derives from `is_single_node: true` as drift on clusters. ([#7577](#7577)) * Stop planning a spurious update for Lakeflow Connect pipelines whose `ingestion_definition.data_staging_options` is filled in by the backend. ([#7603](#7603)) * Retry the root path check in `bundle destroy` on transient errors, so a dropped connection no longer fails the destroy. ([#7597](#7597)) * Use the Lakebase branch when its create request times out but the branch was created, instead of failing the deploy, in the direct deployment engine. ([#7602](#7602)) * Retry transient server errors when deleting resources in the direct deployment engine. ([#7708](#7708)) * Use the Lakebase project when its create request times out but the project was created, instead of failing the deploy with "already exists", in the direct deployment engine. ([#7747](#7747)) * Use the feature, model serving endpoint or dashboard that a create request created despite timing out, instead of failing the deploy, in the direct deployment engine. ([#7757](#7757)) * Use the Lakebase role when its create request fails with a transient error but the role was created, instead of failing the deploy, in the direct deployment engine. ([#7767](#7767)) * Use the MCP service, model service or model provider service that a create request created despite failing with a timeout, a 499 or a reset stream, instead of failing the deploy, in the direct deployment engine. Create recovery for features, serving endpoints and dashboards now also covers 499 and reset streams. ([#7782](#7782)) * Retry app updates and deletes in the direct deployment engine when they fail with 400 BAD_REQUEST because the app compute is `UPDATING`, after waiting for the compute to settle. ([#7606](#7606)) * Enable drift detection for `source_code_path` and `git_source` on apps. ([#6399](#6399)) * `bundle destroy` and `bundle deploy` no longer fail with "internal error, missing in state" when the state holds an empty-ID entry left by an interrupted recreate in an older CLI. ([#7676](#7676)) * Fix a feature whose `full_name` contains uppercase letters being recreated on every deploy. ([#7398](#7398)) * Fix `bundle deployment bind` (and `bundle generate --bind`) failing with `no such node "resources.internal_immutable_snapshots.immutable"` when `workspace.sync_mode` is `immutable`, as reported in [#7306](#7306). ([#7329](#7329)) * Retry creating the deployment lock (without `--force-lock`) when the request fails with a transient server error, times out ("request timed out after ... of inactivity") or loses its connection ("unexpected EOF"). ([#7594](#7594)) * Fix a materialization set member whose `feature_name` contains uppercase letters failing to deploy, or recreating the set on every deploy. ([#7398](#7398)) * Delete the old resource when its name follows a renamed schema. ([#7718](#7718)) * Fix `pipelines dry-run` starting a real run when given a job KEY, and a crash in its shell completion. ([#7672](#7672)) * Rename schemas in place; schemas and volumes follow a renamed catalog or schema. ([#7343](#7343)) * Fix a panic when validating or deploying jobs with local libraries and serverless job clusters. ([#7725](#7725)) * Fix `tgz` artifacts with `git` set producing an empty tarball or failing when the bundle is in a subdirectory of the git repository. ([#7674](#7674)) * Apply `presets.trigger_pause_status` and the development mode pause default to `triggers[]` entries on jobs. ([#7450](#7450)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#7750](#7750)) * Update the Go SDK to [v0.198.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.198.0). ([#7530](#7530)) * Update the Go SDK to [v0.199.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.199.0). ([#7584](#7584)) * Update the Go SDK to [v0.200.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.200.0). ([#7653](#7653)) * Update the Go SDK to [v0.201.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.201.0). ([#7769](#7769))
Stacked on #7467.
Stacked on: aitools: log install-hint telemetry and bound telemetry uploads
Summary
aitools installresolved auth for telemetry in PreRunE, before doing any work, so an unreachable workspace blocked the install for as long as the SDK retried. It now resolves auth in the background while the install runs and waits for it at most 3 s after the command started. Nothing is resolved whenDATABRICKS_CLI_DISABLE_TELEMETRYis set (newtelemetry.Disabled), and a run that logged no event does not wait.The background resolution never refreshes an OAuth token. The process may exit before it finishes, and the server rotates the single-use refresh token, so an abandoned refresh could lose the new one and log the user out. The new
root.SkipTokenRefreshmakesMustWorkspaceClientandMustAccountClientuseauth.CredentialsWithoutRefresh, whosedatabricks-clistrategy passes the newu2m.WithoutRefresh: the cached token is used only while it is valid, and an expired one fails auth, so nothing is uploaded.The shadow command logs through
debugOnlyHandler, so a background warning such as[hostmetadata] failed to fetch host metadata ...shows only with--debugand does not interrupt the install's output or prompts. This relies on root's host-metadata resolver logging through the command's logger (#7390). A panic in the background resolution is recovered and logged at debug level; the install does not fail.Alternatives considered: resolving auth inside
telemetry.Uploadwould serialize it after the install at exit and dropMustAnyClient's account-client fallback and--profilehandling. Bounding a synchronous PreRunE with the SDK'sHTTPTimeoutSeconds/RetryTimeoutSecondswould still add that wait to every install before any work starts and keep the hostmetadata warning in its output; resolving in the background overlaps auth with the install instead.The deprecated
experimental aitools skills installalias runs its own install command, so it now copies that command's context back, and the config resolved for telemetry reaches root's upload (before, the alias never uploaded).Changelog fragment:
.nextchanges/cli/aitools-install-telemetry-auth.md.Evidence
aitools install --skills-only --scope=global --agents=claude-code, skills served locally, fresh HOME and cache per run; before = parent branch binary, after = this branch's head:Tests at head, all pass locally on Linux and in CI on Linux, macOS and Windows:
Merge Danger
Door: two-way
No telemetry schema change, persisted state or config. A revert restores the blocking PreRunE.
Blast Radius:
aitools install.root.SkipTokenRefreshis new and onlyaitools installsets it; without itMustWorkspaceClient,MustAccountClientandPersistentAuth.Tokenbehave as before.DATABRICKS_CLI_TELEMETRY_TIMEOUT, default 3 s), so after a short install the end can take up to about 6 s. No run waits longer for auth than it did before.aitools installrun that logged an event lasts at least 3 s. Runs with nothing to upload (e.g. a flag-validation error) do not wait. The process exits while the SDK is still retrying, before libs/hostmetadata writes its negative-cache entry, so the next run probes the host again.databricks-cliaccess token has expired uploads no install event, because the background resolution does not refresh it. This is on purpose: a refresh that the exit can cut short is what can strand a rotated refresh token. The next command that authenticates refreshes the token as usual. Before, the install refreshed it in PreRunE and uploaded the event.auth_type: with noauth_type(e.g.DATABRICKS_HOSTfrom the environment, or a hand-written profile), the SDK chain skips a strategy whose dry run fails, so an expired token moves on to the strategies afterdatabricks-cli(metadata service, OIDC, Azure, Google). If one succeeds, e.g.azlogged in on an Azure host, the event is uploaded with that identity. The default chain would have refreshed and stopped atdatabricks-cli.databricks-clistrategy,libs/authToOAuthArgumentlooks up host metadata outside the command's logger; SDK warnings logged on its owncontext.Background()(e.g. missingoidc_endpoint) are not filtered either.This pull request and its description were written by Isaac.